Apr 11

I want to take a serious look at serialization and materialization in the coming days. I have some ideas on using contexts for this, similar to what has been done in validations and lazy loading.

  1.  
  2.   class Invoice
  3.     include DataMapper::Resource
  4.     include DataMapper::Serialization
  5.     property :no, String, :serialize => [:invoice_detail,:invoice_summary]
  6.     property :customer, String, :serialize => [:invoice_detail,:invoice_summary]
  7.    
  8.     has :line_items, 0..n, :serialize => [:invoice_detail]
  9.   end
  10.  
  11.   class LineItem
  12.     ..
  13.     ..
  14.     property :description, String, :serialize => [:line_item_detail, :invoice_detail]
  15.   end
  16.  
  17.    ..
  18.    ..
  19.    Invoice.all(Invoice.customer.name => "Joe’s Food").to_xml(:invoice_summary)
  20.    Invoice[23].to_json(:invoice_detail)
  21.  

Wycats had interesting ideas on how to deal with type information about the model. He had a dislike to something such as this:

  1.  
  2.   <zoo>
  3.     <name type="string">Lake Side Zoo</name>
  4.   </zoo>
  5.  

After listening to his proposal, I agree. He suggests that we have the model generate a W3C XML Schema to represent its type information. We can also generate a json version of this.

Another idea is that of pluggable serializers. We could have something like:

  1.  
  2.     to_xml_serializer = DataMapper::Serializers::StandardXMLSerializer
  3.  

Wycats is working on a Lazlo Rest server. Check it out at http://wiki.openlaszlo.org/Laszlo_on_REST

Expect to hear more about this in the next few days.

– until next time –

Apr 2

Last time I wrote about the internals of a proposed ACL plugin. Today I want to focus on the model. How would you define the ACL on the model? What additional methods will be made available on the model?

We should consider the following items:

  • How to set up a default of which resource (self or container) the ACL token is attached to.
  • How to override this default on a per ACL token instance basis. I.e. All on self except READ on container.
  • How to specify the default policy when no ACL is applied to CRUD methods. I.e. GRANT ALL or DENY ALL
  • How will CRUD method ACL tokens be specified
  • How will additional ACL tokens be specified.
  • How will we deal with the CREATE ACL token. It will have to live on the class. There is also the case where resources are “owned”. CREATE would then live on the owner instance.
  • How will we integrate into the finders, and other CRUD methods.
  • What custom methods will we add to the model.
  • Would we want multiple ACL tokens per action? For example reading a resource is allowed if you have MANAGER_READ or EMPLOYEE_READ tokens. (Kind of a silly example, but you get what i mean.) If so, would we want to configure OR’ing or AND’ing the tokens to get a result.

Enough talk! Lets give it a bash.

  1.  
  2. class Organisation
  3.   include DataMapper::Resource
  4.   include SuperDuper::ACL
  5.  
  6.   property :id, Fixnum, :serial => true
  7.   property :name, String
  8.  
  9.   # Start the SuperDuper::ACL configuration
  10.   acl_grant_on :organisation   # When granting store token on self
  11.   acl_deny_all                 # If not explicidly defined, DENY
  12.  
  13.   # Define the tokens we will grant/revoke and protect with
  14.   acl_create   :create, :grant_on => :class
  15.   acl_read     :read
  16.   acl_update   :update
  17.   acl_delete   :delete
  18.  
  19.   # Define a couple of custom tokens
  20.   acl_custom   :liquidate_assets
  21.   acl_custom   :throw_party  
  22. end
  23.  

Lets look at the code above, starting from line 10, acl_grant_on would define where we store the tokens. In this case tokens are stored on the instance. This is one area I want more clarity on. If we had a contact that was contained by an organisation and we were storing the token on the container, what would the syntax look like?

  1.  
  2. class Contact
  3.   include DataMapper::Resource
  4.   include SuperDuper::ACL
  5.  
  6.   property :id, Fixnum, :serial => true
  7.   property :name, String
  8.   property :organisation_id, Fixnum
  9.  
  10.   belongs_to :organisation
  11.  
  12.   # Start the SuperDuper::ACL config
  13.   acl_grant_on ‘contact.organisation’   # looking for a good way to do this.
  14. end
  15.  

On a side note, it is likley that the location of ACL token storage would have to be the instance or an association in the instances association chain. I don’t see being able to store it on some arbitrary object instance.

Moving to line 11 of the Organisation code, acl_deny_all simply says if a CRUD methods ACL token is not explicitly defined, DENY it. The opposite to this would be acl_allow_all which would ALLOW a CRUD method without an ACL token defined. It might be smart to default the resource to acl_deny_all so that you would only need to specify acl_allow_all if you where not happy with the default.

Lines 14-17 set a permission token for each of the CRUD methods. Line 14 shows the overriding of acl_grant_on for a individual ACL instance. In all likelihood acl_create would default to :class and you would not have the include the :grant_on option. An interesting idea on create might be to allow either :class or a named parameter of the constructor to be used. For example:

  1.  
  2. class Organisation
  3.   ..
  4.   acl_create   :create, :grant_on => :owner
  5.   ..
  6.   def initialize(user,owner,opts={})
  7.     ..
  8.   end  
  9. end
  10.  

At this stage I have no idea if/how to intergrate with the constructor. I will ask around! This also needs more thought. Does the above mean when I grant the ACL token to some user on an instance, the token is stored on the instance’s owner (yes), when in fact what I was trying to convey is that before you can create an org with instance as owner, you must have CREATE token on instance (if that makes scene).

Finally, at line 20-21 on the organisation code (top), acl_custom adds a couple of custom ACL tokens to the resource. These are obviously used to determine if the user can sell all the organisations assets and throw a big party with the proceeds.

Well this has turning into a rather long post. I haven’t gotten into the finder & other CRUD methods. They will have to have to wait for a part III.

– until next time –

Apr 1

What about the idea of transparently integrating an access control list into the model via a plugin. Having domain model instance based security is a requirement for some of the projects I work on. The basic idea is to have a GrantedAuthority resource something like:

  1.  
  2.   class GrantedAuthority
  3.     include DataMapper:Resource
  4.     property :id, Fixnum, :serial => true
  5.     property :user_id, Fixnum
  6.     property :resource_id, Fixnum
  7.     property :resource_type, String    
  8.     property :token, String
  9.   end
  10.  

Then you would modify/shadow some of the class & instance methods on the resource that you are protecting, including:

  • Resource class methods
    • Resource#first / Resource#secure_first / Resource#first! / etc
    • Resource#all
    • Resource#get
    • Resource#[]
    • Resource#create
  • Resource instance methods
    • resource#save
    • resource#destroy

This would allow something along the lines of

  1.  
  2.   @contacts = Contact.all!(user,‘READ’, :first_name.like => ‘J%’, … )
  3.  
  4.   # perhaps the READ token is defined on the Resource
  5.   # and you don’t need to specify it here
  6.   @contacts = Contact.all!(user, :first_name.like => ‘J%’, … )
  7.  

The model should “know” how to create the necessary joins to apply the ACL

  1.  
  2.   – #all SQL
  3.   SELECT * FROM contacts
  4.  
  5.   – becomes
  6.   SELECT * FROM contacts WHERE (id IN
  7.       (SELECT resource_id
  8.        FROM granted_authorities
  9.        WHERE user_id = ? AND token = ? AND resource_type = ?
  10.        )
  11.    )
  12.  

This has been fairly simple so far. Lets allow for a more complex case of the resource being queried not being the resource that has the entry in the granted_authorities table. For example, an OrganisationalUnit that contains a number of Contact(s) and the right to ‘READ_CONTACT’ is granted on the OrganisationalUnit. We would need to produce something like this:

  1.  
  2.   SELECT contacts.*, organisational_units.id FROM contacts
  3.   LEFT OUTER JOIN organisational_units ON organisational_units.id = contacts.id
  4.   WHERE (organisational_units.id IN
  5.       (SELECT resource_id
  6.        FROM granted_authorities
  7.        WHERE user_id = ? AND token = ? AND resource_type = ?
  8.        )
  9.    )
  10.  

What does dm-core need to provided for us to build the plugin? We need to support joins to join in the container objects. Dan Kubb is working on that and should have it pretty soon. We must be able to do sub-selects. This feature was push to after the 1.0 release. Fortunately, it was not so hard to implement. I just pushed the required changes to github. You can now do something like:

  1.  
  2.       acl = DataMapper::Query.new(Permission, :user_id => 1, :resource_type => ‘SailBoat’, :token => ‘READ’, :fields => [:resource_id])
  3.       query = DataMapper::Query.new(SailBoat, :port => ‘Cape Town’,:id => acl,:captain.like => ‘J%’)
  4.       boats = @adapter.read_set(repository(:sqlite3),query)    
  5.  

Once I wrap this concept up into a plugin you would do something like:

  1.  
  2.   user = get_current_user
  3.   contacts = Contact.all!(user,‘READ’,:first_name => ‘john’,:city.like => ‘Jo%’)
  4.  
  5.   # or if the token was specified on the Resource
  6.   contacts = Contact.all!(user,:first_name => ‘john’,:city.like => ‘Jo%’)
  7.  

I will write about how you would configure ACL’s on you model/resource in a part II.

– until next time –

Mar 27

When your web application accesses data in the model, it normally does it in couple of different contexts. You are either listing all available resources in a “summary” context, or you are listing the details of a single resource in a “detailed” context. I’m sure you’ll find cases where this is not true, but for the most part it holds true.

Lets take a Contact resource as an example:

  1.  
  2.   class Contact
  3.     include DataMapper::Resource
  4.     property :id, Fixnum, :serial => true
  5.     property :display_name, String
  6.     property :first_name, String
  7.     property :last_name, String
  8.     property :work_phone, String
  9.     property :home_phone, String
  10.     property :email, String
  11.     # ..
  12.     # more properites
  13.     # ..
  14.     property :notes_updated, DateTime
  15.     property :notes_updated_by_id, Fixnum
  16.     property :notes, String
  17.   end
  18.  

You build a RESTful interface to show your new contact resource. Lets say our interface includes:

  • /contacts
  • /contacts/1
  • /contacts/1/notes

This gives you 3 contexts for data access (”summary”, “detailed” and “notes”). Every time someone goes to the /contacts URL your ORM retrieves all the data in the contacts table, maps it into Contact instances and returns an array of these instances. Then you do something like this:

  1.  
  2.   # In your controller
  3.   @contacts = Contact.all()
  4.  
  5.   # And in your view something like
  6.   <% @contacts.each do |contact| %>
  7.     <tr>
  8.       <td><%= contact.display_name %></td>
  9.     </tr>
  10.   <% end %>
  11.  

What a waste! Your ORM loaded 10+ fields from the database. Mapped them to variables in memory and then you only used contact.display_name (maybe contact.id). No problem, your ORM is smart, it can lazy load. So you do something like:

  1.  
  2.   class Contact
  3.     include DataMapper::Resource
  4.     property :id, Fixnum, :serial => true
  5.     property :display_name, String
  6.     property :first_name, String, :lazy => true
  7.     property :last_name, String, :lazy => true
  8.     property :work_phone, String, :lazy => true
  9.     property :home_phone, String, :lazy => true
  10.     property :email, String, :lazy => true
  11.     # ..
  12.     # more properites
  13.     # ..
  14.     property :notes_updated, DateTime, :lazy => true
  15.     property :notes_updated_by_id, Fixnum, :lazy => true
  16.     property :notes, String, :lazy => true
  17.   end
  18.  

That works great. When a user list all your contacts, the ORM only pulls out the id and display_name field. Now a user hits /contacts/1 URL.

The problem is your detailed view is a bit more verbose and outputs many more properties. All the lazy properties are loaded on first access. This means one query per lazy property read. Not great. Whats worse is if another part of your code does a Contact.all() and then iterate over the array accessing lazy properties.

The solution. In the soon to be release 0.9.0 of DataMapper lazy loaded properties are going to have contexts. When you read the first lazy property in a context, all the properties in that context for all the instances are loaded in one query. You can do something like this:

  1.  
  2.   class Contact
  3.     include DataMapper::Resource
  4.     property :id, Fixnum, :serial => true
  5.     property :display_name, String
  6.     property :first_name, String, :lazy => [:detailed,:notes]
  7.     property :last_name, String, :lazy => [:detailed,:notes]
  8.     property :work_phone, String, :lazy => [:detailed]
  9.     property :home_phone, String, :lazy => [:detailed]
  10.     property :email, String, :lazy => [:detailed]
  11.     # ..
  12.     # more properites
  13.     # ..
  14.     property :notes_updated, DateTime, :lazy => [:notes]
  15.     property :notes_updated_by_id, Fixnum, :lazy => [:notes]
  16.     property :notes, String, :lazy => [:notes]
  17.   end
  18.  

– until next time –

Mar 26
15 Things To Write About
icon1 guyvdb | icon2 Ruby | icon4 03 26th, 2008| icon3No Comments »

Starting a blog is all about having something to say. Something that others may find interesting. Wanting to be successful at blogging, I regularly need something interesting to say. Some homework, come up with 15 10 on-topic ideas that I can write a few paragraphs on. In no particular order, here they are:

  1. DataMapper 0.9.0 Validations
  2. Context Based Recursive Serialization for DataMapper Resources
  3. Access Control List Plugin for Merb/DataMapper
  4. Using the English Gem
  5. DataMapper/Merb Class Diagram Creator - ala RailRoad
  6. Object/Relational Mismatch - Strategies for Inheritance
  7. GEdit - A Powerful Ruby Development Environment
  8. Role Player - Using Polymorphic Association to put Everything into a Role
  9. DataMapper LazyLoading with Contexts
  10. ? (Still thinking)

Over the next ~10 days, you know what I am going to write about.

– until next time –

Mar 25
Welcome
icon1 guyvdb | icon2 Uncategorized | icon4 03 25th, 2008| icon3No Comments »

About Me

I live in Johannesburg, South Africa. I work in the IT sector. For the last few years, most of my time has been devoted to a project for a physical security equipment manufacture. The project is made up of an embedded micro-processor (ARM7), a home grown embedded OS and a web application. It governs security at multi-dwelling residential complexes.

Current Work

The current version of the product was written in C (on the embedded hardware) and Java (for the web application). The next release, which I am busy with, is a complete re-write. We are moving to a preemptive multi-tasking kernel on the embedded hardware (still in C of course), a RESTful API (written in my newly found favorite language Ruby) on the server side and for the browser, a RIA client (written in ActionScript/Flash).

Starting a Blog

So that’s a little bit about me. Now for the what and why of this blog. This is not my first attempt at starting a blog. I have begun half heartedly before. Slowly petering out. This time, I have invested in a domain name and setup the blog on my own server. I hope this will build some momentum which will spur me onward.

Reading about “how to start a successful blog” on the plethora of blogs that dish out that kind of advice, I have noted that I should:

  • take care of all the technical stuff
  • decide what to blog about
  • get a first post out the door
  • get into the habit of blogging
  • promote your blog
  • practice, practice, practice
  • ignore the lack of readers :)

I have taken care of all the technical stuff. I have a domain name, the server is set up, everything is ready to go. Next, to pick a subject. What am I going to blog about. Ruby to start. Maybe a touch of embedded development and C. Right now I am getting the first post out the door.

Only time will tell - do I get into the habit of blogging? I already have the subject for my next entry. Pick 10 subjects related to Ruby that I want to blog about.

– until next time –