Writing Property Subclasses

The Property class is designed to be subclassed. However, it is normally easier to subclass an existing Property subclass.

All special Property attributes, even those considered 'public', have names starting with an underscore. This is because StructuredProperty uses the non-underscore attribute namespace to refer to nested Property names; this is essential for specifying queries on subproperties.

The Property class and its predefined subclasses allow subclassing using composable (or stackable) validation and conversion APIs. These require some terminology definitions:

• A user value is a value such as would be set and accessed by the application code using standard attributes on the entity.
• A base value is a value such as would be serialized to and deserialized from the Datastore.

A Property subclass that implements a specific transformation between user values and serializable values should implement two methods, _to_base_type() and _from_base_type() . These should notcall their super() method. This is what is meant by composable (or stackable) APIs.

The API supports stacking classes with ever more sophisticated user-base conversions: the user-to-base conversion goes from more sophisticated to less sophisticated, while the base-to-user conversion goes from less sophisticated to more sophisticated. For example, see the relationship between BlobProperty , TextProperty , and StringProperty . For example, TextProperty inherits from BlobProperty ; its code is pretty simple because it inherits most of the behavior it needs.

In addition to _to_base_type() and _from_base_type() , the _validate() method is also a composable API.

The validation API distinguishes between lax and strict user values. The set of lax values is a superset of the set of strict values. The _validate() method takes a lax value and if necessary converts it to a strict value. This means that when setting the property value, lax values are accepted, while when getting the property value, only strict values will be returned. If no conversion is needed, _validate() may return None. If the argument is outside the set of accepted lax values, _validate() should raise an exception, preferably TypeError or datastore_errors.BadValueError .

The _validate() , _to_base_type() , and _from_base_type() do not need to handle:

• None : They will not be called with None (and if they return None, this means that the value does not need conversion).
• Repeated values: The infrastructure takes care of calling _from_base_type() or _to_base_type() for each list item in a repeated value.
• Distinguishing user values from base values: The infrastructure handles this by calling the composable APIs.
• Comparisons: The comparison operations call _to_base_type() on their operand.
• Distinguishing between user and base values: the infrastructure guarantees that _from_base_type() will be called with an (unwrapped) base value, and that _to_base_type() will be called with a user value.

For example, suppose you need to store really long integers. The standard IntegerProperty only supports (signed) 64-bit integers. Your property might store a longer integer as a string; it would be good to have the property class handle the conversion. An application using your property class might look something like

  from 
  
 datetime 
  
 import 
 date 
 import 
  
 my_models 
 

  class 
  
 MyModel 
 ( 
 ndb 
 . 
 Model 
 ): 
 name 
 = 
 ndb 
 . 
 StringProperty 
 () 
 abc 
 = 
 LongIntegerProperty 
 ( 
 default 
 = 
 0 
 ) 
 xyz 
 = 
 LongIntegerProperty 
 ( 
 repeated 
 = 
 True 
 ) 
 

  # Create an entity and write it to the Datastore. 
 entity 
 = 
 my_models 
 . 
 MyModel 
 ( 
 name 
 = 
 'booh' 
 , 
 xyz 
 = 
 [ 
 10 
 ** 
 100 
 , 
 6 
 ** 
 666 
 ]) 
 assert 
 entity 
 . 
 abc 
 == 
 0 
 key 
 = 
 entity 
 . 
 put 
 () 
 

  # Read an entity back from the Datastore and update it. 
 entity 
 = 
 key 
 . 
 get 
 () 
 entity 
 . 
 abc 
 += 
 1 
 entity 
 . 
 xyz 
 . 
 append 
 ( 
 entity 
 . 
 abc 
 // 
 3 
 ) 
 entity 
 . 
 put 
 () 
 

  # Query for a MyModel entity whose xyz contains 6**666. 
 # (NOTE: using ordering operations don't work, but == does.) 
 results 
 = 
 my_models 
 . 
 MyModel 
 . 
 query 
 ( 
 my_models 
 . 
 MyModel 
 . 
 xyz 
 == 
 6 
 ** 
 666 
 ) 
 . 
 fetch 
 ( 
 10 
 ) 
 

This looks simple and straightforward. It also demonstrates the use of some standard property options (default, repeated); as the author of LongIntegerProperty , you will be glad to hear you don't have to write any "boilerplate" to get those working. It's easier to define a subclass of another property, for example:

  class 
  
 LongIntegerProperty 
 ( 
 ndb 
 . 
 StringProperty 
 ): 
 def 
  
 _validate 
 ( 
 self 
 , 
 value 
 ): 
 if 
 not 
 isinstance 
 ( 
 value 
 , 
 ( 
 int 
 , 
 long 
 )): 
 raise 
 TypeError 
 ( 
 'expected an integer, got 
 %s 
 ' 
 % 
 repr 
 ( 
 value 
 )) 
 def 
  
 _to_base_type 
 ( 
 self 
 , 
 value 
 ): 
 return 
 str 
 ( 
 value 
 ) 
 # Doesn't matter if it's an int or a long 
 def 
  
 _from_base_type 
 ( 
 self 
 , 
 value 
 ): 
 return 
 long 
 ( 
 value 
 ) 
 # Always return a long 
 

When you set a property value on an entity, e.g. ent.abc = 42 , your _validate() method is called, and (if it doesn't raise an exception) the value is stored on the entity. When you write the entity to the Datastore, your _to_base_type() method is called, converting the value to the string. Then that value is serialized by the base class, StringProperty . The inverse chain of events happens when the entity is read back from the Datastore. The StringProperty and Property classes together take care of the other details, such as serializing the and deserializing the string, setting the default, and handling repeated property values.

In this example, supporting inequalities (i.e. queries using <, <=, >, >=) requires more work. The following example implementation imposes a maximum size of integer and stores values as fixed-length strings:

  class 
  
 BoundedLongIntegerProperty 
 ( 
 ndb 
 . 
 StringProperty 
 ): 
 def 
  
 __init__ 
 ( 
 self 
 , 
 bits 
 , 
 ** 
 kwds 
 ): 
 assert 
 isinstance 
 ( 
 bits 
 , 
 int 
 ) 
 assert 
 bits 
> 0 
 and 
 bits 
 % 
 4 
 == 
 0 
 # Make it simple to use hex 
 super 
 ( 
 BoundedLongIntegerProperty 
 , 
 self 
 ) 
 . 
 __init__ 
 ( 
 ** 
 kwds 
 ) 
 self 
 . 
 _bits 
 = 
 bits 
 def 
  
 _validate 
 ( 
 self 
 , 
 value 
 ): 
 assert 
 - 
 ( 
 2 
 ** 
 ( 
 self 
 . 
 _bits 
 - 
 1 
 )) 
< = 
 value 
< 2 
 ** 
 ( 
 self 
 . 
 _bits 
 - 
 1 
 ) 
 def 
  
 _to_base_type 
 ( 
 self 
 , 
 value 
 ): 
 # convert from signed -> unsigned 
 if 
 value 
< 0 
 : 
 value 
 += 
 2 
 ** 
 self 
 . 
 _bits 
 assert 
 0 
< = 
 value 
< 2 
 ** 
 self 
 . 
 _bits 
 # Return number as a zero-padded hex string with correct number of 
 # digits: 
 return 
 ' 
 %0*x 
 ' 
 % 
 ( 
 self 
 . 
 _bits 
 // 
 4 
 , 
 value 
 ) 
 def 
  
 _from_base_type 
 ( 
 self 
 , 
 value 
 ): 
 value 
 = 
 int 
 ( 
 value 
 , 
 16 
 ) 
 if 
 value 
> = 
 2 
 ** 
 ( 
 self 
 . 
 _bits 
 - 
 1 
 ): 
 value 
 -= 
 2 
 ** 
 self 
 . 
 _bits 
 return 
 value 
 

This can be used in the same way as LongIntegerProperty except that you must pass the number of bits to the property constructor, e.g. BoundedLongIntegerProperty(1024) .

You can subclass other property types in similar ways.

This approach also works for storing structured data. Suppose you have a FuzzyDate Python class that represents a date range; it uses fields first and last to store the date range's beginning and end:

  from 
  
 datetime 
  
 import 
 date 
 

  class 
  
 FuzzyDate 
 ( 
 object 
 ): 
 def 
  
 __init__ 
 ( 
 self 
 , 
 first 
 , 
 last 
 = 
 None 
 ): 
 assert 
 isinstance 
 ( 
 first 
 , 
 date 
 ) 
 assert 
 last 
 is 
 None 
 or 
 isinstance 
 ( 
 last 
 , 
 date 
 ) 
 self 
 . 
 first 
 = 
 first 
 self 
 . 
 last 
 = 
 last 
 or 
 first 
 

You can create a FuzzyDateProperty that derives from StructuredProperty . Unfortunately, the latter doesn't work with plain old Python classes; it needs a Model subclass. So define a Model subclass as an intermediate representation;

  class 
  
 FuzzyDateModel 
 ( 
 ndb 
 . 
 Model 
 ): 
 first 
 = 
 ndb 
 . 
 DateProperty 
 () 
 last 
 = 
 ndb 
 . 
 DateProperty 
 () 
 

Next, construct a subclass of StructuredProperty that hardcodes the modelclass argument to be FuzzyDateModel , and defines _to_base_type() and _from_base_type() methods to convert between FuzzyDate and FuzzyDateModel :

  class 
  
 FuzzyDateProperty 
 ( 
 ndb 
 . 
 StructuredProperty 
 ): 
 def 
  
 __init__ 
 ( 
 self 
 , 
 ** 
 kwds 
 ): 
 super 
 ( 
 FuzzyDateProperty 
 , 
 self 
 ) 
 . 
 __init__ 
 ( 
 FuzzyDateModel 
 , 
 ** 
 kwds 
 ) 
 def 
  
 _validate 
 ( 
 self 
 , 
 value 
 ): 
 assert 
 isinstance 
 ( 
 value 
 , 
 FuzzyDate 
 ) 
 def 
  
 _to_base_type 
 ( 
 self 
 , 
 value 
 ): 
 return 
 FuzzyDateModel 
 ( 
 first 
 = 
 value 
 . 
 first 
 , 
 last 
 = 
 value 
 . 
 last 
 ) 
 def 
  
 _from_base_type 
 ( 
 self 
 , 
 value 
 ): 
 return 
 FuzzyDate 
 ( 
 value 
 . 
 first 
 , 
 value 
 . 
 last 
 ) 
 

An application might use this class like so:

  class 
  
 HistoricPerson 
 ( 
 ndb 
 . 
 Model 
 ): 
 name 
 = 
 ndb 
 . 
 StringProperty 
 () 
 birth 
 = 
 FuzzyDateProperty 
 () 
 death 
 = 
 FuzzyDateProperty 
 () 
 # Parallel lists: 
 event_dates 
 = 
 FuzzyDateProperty 
 ( 
 repeated 
 = 
 True 
 ) 
 event_names 
 = 
 ndb 
 . 
 StringProperty 
 ( 
 repeated 
 = 
 True 
 ) 
 

  columbus 
 = 
 my_models 
 . 
 HistoricPerson 
 ( 
 name 
 = 
 'Christopher Columbus' 
 , 
 birth 
 = 
 my_models 
 . 
 FuzzyDate 
 ( 
 date 
 ( 
 1451 
 , 
 8 
 , 
 22 
 ), 
 date 
 ( 
 1451 
 , 
 10 
 , 
 31 
 )), 
 death 
 = 
 my_models 
 . 
 FuzzyDate 
 ( 
 date 
 ( 
 1506 
 , 
 5 
 , 
 20 
 )), 
 event_dates 
 = 
 [ 
 my_models 
 . 
 FuzzyDate 
 ( 
 date 
 ( 
 1492 
 , 
 1 
 , 
 1 
 ), 
 date 
 ( 
 1492 
 , 
 12 
 , 
 31 
 ))], 
 event_names 
 = 
 [ 
 'Discovery of America' 
 ]) 
 columbus 
 . 
 put 
 () 
 # Query for historic people born no later than 1451. 
 results 
 = 
 my_models 
 . 
 HistoricPerson 
 . 
 query 
 ( 
 my_models 
 . 
 HistoricPerson 
 . 
 birth 
 . 
 last 
< = 
 date 
 ( 
 1451 
 , 
 12 
 , 
 31 
 )) 
 . 
 fetch 
 () 
 

Suppose you want to accept plain date objects in addition to FuzzyDate objects as the values for FuzzyDateProperty . To do this, modify the _validate() method as follows:

  def 
  
 _validate 
 ( 
 self 
 , 
 value 
 ): 
 if 
 isinstance 
 ( 
 value 
 , 
 date 
 ): 
 return 
 FuzzyDate 
 ( 
 value 
 ) 
 # Must return the converted value! 
 # Otherwise, return None and leave validation to the base class 
 

You could instead subclass FuzzyDateProperty as follows (assuming FuzzyDateProperty._validate() is as shown above).

  class 
  
 MaybeFuzzyDateProperty 
 ( 
 FuzzyDateProperty 
 ): 
 def 
  
 _validate 
 ( 
 self 
 , 
 value 
 ): 
 if 
 isinstance 
 ( 
 value 
 , 
 date 
 ): 
 return 
 FuzzyDate 
 ( 
 value 
 ) 
 # Must return the converted value! 
 # Otherwise, return None and leave validation to the base class 
 

When you assign a value to a MaybeFuzzyDateProperty field, both MaybeFuzzyDateProperty._validate() and FuzzyDateProperty._validate() are invoked, in that order. The same applies to _to_base_type() and _from_base_type() : the methods in in superclass and subclass are implicitly combined. (Don't use super to control inherited behavior for this. For these three methods, the interaction is subtle and super doesn't do what you want.)