Guyvdb.info

Serialization and Materialization

guyvdb — Fri, 11 Apr 2008 14:31:38 +0000

I want to take a serious look at serialization and materialization in the coming days. I have some ideas on using contexts for this, similar to what has been done in validations and lazy loading.

class Invoice
include DataMapper::Resource
include DataMapper::Serialization
property :no, String, :serialize => [:invoice_detail,:invoice_summary]
property :customer, String, :serialize => [:invoice_detail,:invoice_summary]
has :line_items, 0..n, :serialize => [:invoice_detail]
end
class LineItem
..
..
property :description, String, :serialize => [:line_item_detail, :invoice_detail]
end
..
..
Invoice.all(Invoice.customer.name => "Joe’s Food").to_xml(:invoice_summary)
Invoice[23].to_json(:invoice_detail)

Wycats had interesting ideas on how to deal with type information about the model. He had a dislike to something such as this:

>
type="string">Lake Side Zoo>
>

After listening to his proposal, I agree. He suggests that we have the model generate a W3C XML Schema to represent its type information. We can also generate a json version of this.

Another idea is that of pluggable serializers. We could have something like:

to_xml_serializer = DataMapper::Serializers::StandardXMLSerializer

Wycats is working on a Lazlo Rest server. Check it out at http://wiki.openlaszlo.org/Laszlo_on_REST

Expect to hear more about this in the next few days.

– until next time –

Builing an ACL Plugin for DataMapper - Part II

guyvdb — Wed, 02 Apr 2008 10:10:26 +0000

Last time I wrote about the internals of a proposed ACL plugin. Today I want to focus on the model. How would you define the ACL on the model? What additional methods will be made available on the model?

We should consider the following items:

How to set up a default of which resource (self or container) the ACL token is attached to.
How to override this default on a per ACL token instance basis. I.e. All on self except READ on container.
How to specify the default policy when no ACL is applied to CRUD methods. I.e. GRANT ALL or DENY ALL
How will CRUD method ACL tokens be specified
How will additional ACL tokens be specified.
How will we deal with the CREATE ACL token. It will have to live on the class. There is also the case where resources are “owned”. CREATE would then live on the owner instance.
How will we integrate into the finders, and other CRUD methods.
What custom methods will we add to the model.
Would we want multiple ACL tokens per action? For example reading a resource is allowed if you have MANAGER_READ or EMPLOYEE_READ tokens. (Kind of a silly example, but you get what i mean.) If so, would we want to configure OR’ing or AND’ing the tokens to get a result.

Enough talk! Lets give it a bash.

class Organisation
include DataMapper::Resource
include SuperDuper::ACL
property :id, Fixnum, :serial => true
property :name, String
# Start the SuperDuper::ACL configuration
acl_grant_on :organisation # When granting store token on self
acl_deny_all # If not explicidly defined, DENY
# Define the tokens we will grant/revoke and protect with
acl_create :create, :grant_on => :class
acl_read :read
acl_update :update
acl_delete :delete
# Define a couple of custom tokens
acl_custom :liquidate_assets
acl_custom :throw_party
end

Lets look at the code above, starting from line 10, acl_grant_on would define where we store the tokens. In this case tokens are stored on the instance. This is one area I want more clarity on. If we had a contact that was contained by an organisation and we were storing the token on the container, what would the syntax look like?

class Contact
include DataMapper::Resource
include SuperDuper::ACL
property :id, Fixnum, :serial => true
property :name, String
property :organisation_id, Fixnum
belongs_to :organisation
# Start the SuperDuper::ACL config
acl_grant_on ‘contact.organisation’ # looking for a good way to do this.
end

On a side note, it is likley that the location of ACL token storage would have to be the instance or an association in the instances association chain. I don’t see being able to store it on some arbitrary object instance.

Moving to line 11 of the Organisation code, acl_deny_all simply says if a CRUD methods ACL token is not explicitly defined, DENY it. The opposite to this would be acl_allow_all which would ALLOW a CRUD method without an ACL token defined. It might be smart to default the resource to acl_deny_all so that you would only need to specify acl_allow_all if you where not happy with the default.

Lines 14-17 set a permission token for each of the CRUD methods. Line 14 shows the overriding of acl_grant_on for a individual ACL instance. In all likelihood acl_create would default to :class and you would not have the include the :grant_on option. An interesting idea on create might be to allow either :class or a named parameter of the constructor to be used. For example:

class Organisation
..
acl_create :create, :grant_on => :owner
..
def initialize(user,owner,opts={})
..
end
end

At this stage I have no idea if/how to intergrate with the constructor. I will ask around! This also needs more thought. Does the above mean when I grant the ACL token to some user on an instance, the token is stored on the instance’s owner (yes), when in fact what I was trying to convey is that before you can create an org with instance as owner, you must have CREATE token on instance (if that makes scene).

Finally, at line 20-21 on the organisation code (top), acl_custom adds a couple of custom ACL tokens to the resource. These are obviously used to determine if the user can sell all the organisations assets and throw a big party with the proceeds.

Well this has turning into a rather long post. I haven’t gotten into the finder & other CRUD methods. They will have to have to wait for a part III.

– until next time –

Builing an Access Control List Plugin for DataMapper - Part I

guyvdb — Tue, 01 Apr 2008 19:50:48 +0000

What about the idea of transparently integrating an access control list into the model via a plugin. Having domain model instance based security is a requirement for some of the projects I work on. The basic idea is to have a GrantedAuthority resource something like:

class GrantedAuthority
include DataMapper:Resource
property :id, Fixnum, :serial => true
property :user_id, Fixnum
property :resource_id, Fixnum
property :resource_type, String
property :token, String
end

Then you would modify/shadow some of the class & instance methods on the resource that you are protecting, including:

Resource class methods
- Resource#first / Resource#secure_first / Resource#first! / etc
- Resource#all
- Resource#get
- Resource#[]
- Resource#create
Resource instance methods
- resource#save
- resource#destroy

This would allow something along the lines of

@contacts = Contact.all!(user,‘READ’, :first_name.like => ‘J%’, … )
# perhaps the READ token is defined on the Resource
# and you don’t need to specify it here
@contacts = Contact.all!(user, :first_name.like => ‘J%’, … )

The model should “know” how to create the necessary joins to apply the ACL

– #all SQL
SELECT * FROM contacts
– becomes
SELECT * FROM contacts WHERE (id IN
(SELECT resource_id
FROM granted_authorities
WHERE user_id = ? AND token = ? AND resource_type = ?
)
)

This has been fairly simple so far. Lets allow for a more complex case of the resource being queried not being the resource that has the entry in the granted_authorities table. For example, an OrganisationalUnit that contains a number of Contact(s) and the right to ‘READ_CONTACT’ is granted on the OrganisationalUnit. We would need to produce something like this:

SELECT contacts.*, organisational_units.id FROM contacts
LEFT OUTER JOIN organisational_units ON organisational_units.id = contacts.id
WHERE (organisational_units.id IN
(SELECT resource_id
FROM granted_authorities
WHERE user_id = ? AND token = ? AND resource_type = ?
)
)

What does dm-core need to provided for us to build the plugin? We need to support joins to join in the container objects. Dan Kubb is working on that and should have it pretty soon. We must be able to do sub-selects. This feature was push to after the 1.0 release. Fortunately, it was not so hard to implement. I just pushed the required changes to github. You can now do something like:

acl = DataMapper::Query.new(Permission, :user_id => 1, :resource_type => ‘SailBoat’, :token => ‘READ’, :fields => [:resource_id])
query = DataMapper::Query.new(SailBoat, :port => ‘Cape Town’,:id => acl,:captain.like => ‘J%’)
boats = @adapter.read_set(repository(:sqlite3),query)

Once I wrap this concept up into a plugin you would do something like:

user = get_current_user
contacts = Contact.all!(user,‘READ’,:first_name => ‘john’,:city.like => ‘Jo%’)
# or if the token was specified on the Resource
contacts = Contact.all!(user,:first_name => ‘john’,:city.like => ‘Jo%’)

I will write about how you would configure ACL’s on you model/resource in a part II.

– until next time –

Lazy Loading Properties in DataMapper

guyvdb — Thu, 27 Mar 2008 20:42:43 +0000

When your web application accesses data in the model, it normally does it in couple of different contexts. You are either listing all available resources in a “summary” context, or you are listing the details of a single resource in a “detailed” context. I’m sure you’ll find cases where this is not true, but for the most part it holds true.

Lets take a Contact resource as an example:

class Contact
include DataMapper::Resource
property :id, Fixnum, :serial => true
property :display_name, String
property :first_name, String
property :last_name, String
property :work_phone, String
property :home_phone, String
property :email, String
# ..
# more properites
# ..
property :notes_updated, DateTime
property :notes_updated_by_id, Fixnum
property :notes, String
end

You build a RESTful interface to show your new contact resource. Lets say our interface includes:

/contacts
/contacts/1
/contacts/1/notes

This gives you 3 contexts for data access (”summary”, “detailed” and “notes”). Every time someone goes to the /contacts URL your ORM retrieves all the data in the contacts table, maps it into Contact instances and returns an array of these instances. Then you do something like this:

# In your controller
@contacts = Contact.all()
# And in your view something like
<% @contacts.each do |contact| %>
<%= contact.display_name %>
<% end %>

What a waste! Your ORM loaded 10+ fields from the database. Mapped them to variables in memory and then you only used contact.display_name (maybe contact.id). No problem, your ORM is smart, it can lazy load. So you do something like:

class Contact
include DataMapper::Resource
property :id, Fixnum, :serial => true
property :display_name, String
property :first_name, String, :lazy => true
property :last_name, String, :lazy => true
property :work_phone, String, :lazy => true
property :home_phone, String, :lazy => true
property :email, String, :lazy => true
# ..
# more properites
# ..
property :notes_updated, DateTime, :lazy => true
property :notes_updated_by_id, Fixnum, :lazy => true
property :notes, String, :lazy => true
end

That works great. When a user list all your contacts, the ORM only pulls out the id and display_name field. Now a user hits /contacts/1 URL.

The problem is your detailed view is a bit more verbose and outputs many more properties. All the lazy properties are loaded on first access. This means one query per lazy property read. Not great. Whats worse is if another part of your code does a Contact.all() and then iterate over the array accessing lazy properties.

The solution. In the soon to be release 0.9.0 of DataMapper lazy loaded properties are going to have contexts. When you read the first lazy property in a context, all the properties in that context for all the instances are loaded in one query. You can do something like this:

class Contact
include DataMapper::Resource
property :id, Fixnum, :serial => true
property :display_name, String
property :first_name, String, :lazy => [:detailed,:notes]
property :last_name, String, :lazy => [:detailed,:notes]
property :work_phone, String, :lazy => [:detailed]
property :home_phone, String, :lazy => [:detailed]
property :email, String, :lazy => [:detailed]
# ..
# more properites
# ..
property :notes_updated, DateTime, :lazy => [:notes]
property :notes_updated_by_id, Fixnum, :lazy => [:notes]
property :notes, String, :lazy => [:notes]
end

– until next time –

15 Things To Write About

guyvdb — Wed, 26 Mar 2008 05:52:52 +0000

Starting a blog is all about having something to say. Something that others may find interesting. Wanting to be successful at blogging, I regularly need something interesting to say. Some homework, come up with 15 10 on-topic ideas that I can write a few paragraphs on. In no particular order, here they are:

DataMapper 0.9.0 Validations
Context Based Recursive Serialization for DataMapper Resources
Access Control List Plugin for Merb/DataMapper
Using the English Gem
DataMapper/Merb Class Diagram Creator - ala RailRoad
Object/Relational Mismatch - Strategies for Inheritance
GEdit - A Powerful Ruby Development Environment
Role Player - Using Polymorphic Association to put Everything into a Role
DataMapper LazyLoading with Contexts
? (Still thinking)

Over the next ~10 days, you know what I am going to write about.

– until next time –

Welcome

guyvdb — Tue, 25 Mar 2008 18:55:12 +0000

About Me

I live in Johannesburg, South Africa. I work in the IT sector. For the last few years, most of my time has been devoted to a project for a physical security equipment manufacture. The project is made up of an embedded micro-processor (ARM7), a home grown embedded OS and a web application. It governs security at multi-dwelling residential complexes.

Current Work

The current version of the product was written in C (on the embedded hardware) and Java (for the web application). The next release, which I am busy with, is a complete re-write. We are moving to a preemptive multi-tasking kernel on the embedded hardware (still in C of course), a RESTful API (written in my newly found favorite language Ruby) on the server side and for the browser, a RIA client (written in ActionScript/Flash).

Starting a Blog

So that’s a little bit about me. Now for the what and why of this blog. This is not my first attempt at starting a blog. I have begun half heartedly before. Slowly petering out. This time, I have invested in a domain name and setup the blog on my own server. I hope this will build some momentum which will spur me onward.

Reading about “how to start a successful blog” on the plethora of blogs that dish out that kind of advice, I have noted that I should:

take care of all the technical stuff
decide what to blog about
get a first post out the door
get into the habit of blogging
promote your blog
practice, practice, practice
ignore the lack of readers

I have taken care of all the technical stuff. I have a domain name, the server is set up, everything is ready to go. Next, to pick a subject. What am I going to blog about. Ruby to start. Maybe a touch of embedded development and C. Right now I am getting the first post out the door.

Only time will tell - do I get into the habit of blogging? I already have the subject for my next entry. Pick 10 subjects related to Ruby that I want to blog about.

– until next time –