Commit 2fec093a authored by David Michael Brown's avatar David Michael Brown Committed by GitHub
Browse files

Merge pull request #13 from davebshow/vertex_properties

WIP adding support for vertex properties
parents fd1d6dcd 64c2da4f
## Why davebshow/goblin???
Let's face it, ZEROFAIL/goblin has problems. It's not really anyone's fault, it is just a fact of life. The mogwai codebase is monumental and it makes heavy use of metaprogramming, instrumentation, class inheritance, global variables and configuration, and third-party libraries. This results in opaque code that is hard to read and maintain, resulting in less community interest and more work for developers looking to make improvements. My port to TinkerPop3 just made things worse, as it introduced a level of callback hell only know to the most hardcore JavaScript developers. At this point, while ZEROFAIL/goblin basically works, implementing new functionality is a daunting task, and simple debugging and reasoning have become serious work. I was trying to think how to remedy these problems and rescue goblin from an ugly, bloated fate as we move forward adding new functionality. Then, it dawned on me: all I need to do is drag that whole folder to the virtual trash bin on my desktop and start from scratch.
Developers note:
The original Goblin was a TinkerPop 3 ready port of Cody Lee's mogwai, an excellent library that had been developed for use with pre-TinkerPop 3 versions of Titan. We designed Goblin to provide asynchronous programming abstractions that would work using any version of Python 2.7 + with a variety of asynchronous I/O libraries (Tornado, Asyncio, Trollius). While in theory this was great, we found that in our effort to promote compatibility we lost out on many of the features the newer Python versions provide to help developers deal with asynchronous programming. Our code base became large and made heavy use of callbacks, and nearly all methods and functions returned some sort of `Future`. This created both a clunky user API, and a code base that was difficult to reason about and maintain.
But, wait, a whole new OGM from scratch...? Well, yes and no. Borrowing a few concepts from SQLAlchemy and using what I've learned from working on previous Python software targeting the Gremlin Server, I was able to piece together a fully functioning system in just a few hours of work and less than 1/10 the code. So, here is my *community prototype* OGM contribution to the Python TinkerPop3 ecosystem.
So, we decided to rewrite Goblin from scratch...
## Features
......@@ -15,90 +16,3 @@ But, wait, a whole new OGM from scratch...? Well, yes and no. Borrowing a few co
7. Fully extensible data type system
8. Descriptor based property assignment
9. And more...!
### Install
```
$ pip install git+https://github.com/davebshow/goblin.git
```
### Create/update elements
```python
import asyncio
from goblin.api import create_engine, Vertex, Edge
from goblin.properties import Property, String
class TestVertex(Vertex):
__label__ = 'test_vertex'
name = Property(String)
notes = Property(String, initval='N/A')
class TestEdge(Edge):
__label__ = 'test_edge'
notes = Property(String, initval='N/A')
loop = asyncio.get_event_loop()
engine = loop.run_until_complete(
create_engine("http://localhost:8182/", loop))
async def create():
session = engine.session()
leif = TestVertex()
leif.name = 'leifur'
jon = TestVertex()
jon.name = 'jonathan'
works_for = TestEdge()
works_for.source = jon
works_for.target = leif
assert works_for.notes == 'N/A'
works_for.notes = 'zerofail'
session.add(leif, jon, works_for)
await session.flush()
print(leif.name, leif.id, jon.name, jon.id,
works_for.notes, works_for.id)
leif.name = 'leif'
session.add(leif)
await session.flush()
print(leif.name, leif.id)
loop.run_until_complete(create())
# leifur 0 jonathan 3 zerofail 6
# leif 0
```
### Query the db:
```python
async def query():
session = engine.session()
stream = await session.query(TestVertex).all()
async for msg in stream:
print(msg)
loop.run_until_complete(query())
# [<__main__.TestVertex object at 0x7f46d833e588>, <__main__.TestVertex object at 0x7f46d833e780>]
```
### See how objects map to the db:
```python
TestVertex.__mapping__
# <Mapping(type=vertex, label=test_vertex, properties=[
# {'db_name': 'test_vertex__name', 'ogm_name': 'name', 'data_type': <class 'goblin.properties.String'>},
# {'db_name': 'test_vertex__notes', 'ogm_name': 'notes', 'data_type': <class 'goblin.properties.String'>}])
```
### Close the engine
```python
loop.run_until_complete(engine.close())
```
......@@ -15,6 +15,7 @@
# You should have received a copy of the GNU Affero General Public License
# along with Goblin. If not, see <http://www.gnu.org/licenses/>.
from goblin.element import Vertex, Edge, VertexProperty
from goblin.app import create_app, Goblin
from goblin.cardinality import Cardinality
from goblin.element import Vertex, Edge, VertexProperty
from goblin.properties import Property, String
......@@ -16,6 +16,12 @@
# along with Goblin. If not, see <http://www.gnu.org/licenses/>.
import abc
import logging
from goblin import cardinality, manager, exception
logger = logging.getLogger(__name__)
class DataType(abc.ABC):
......@@ -44,11 +50,37 @@ class DataType(abc.ABC):
@abc.abstractmethod
def to_ogm(self, val):
"""Convert property value to a Python compatible format"""
try:
self.validate(val)
except exception.ValidationError:
logger.warning(
"DB val {} Fails OGM validation for {}".format(val, self))
return val
def validate_vertex_prop(self, val, card, vertex_prop, data_type):
if card == cardinality.Cardinality.list:
if isinstance(val, list):
val = val
elif isinstance(val, (set, tuple)):
val = list(val)
else:
val = [val]
val = manager.ListVertexPropertyManager(
data_type,
vertex_prop,
card,
[vertex_prop(data_type, val=self.validate(v), card=card)
for v in val])
elif card == cardinality.Cardinality.set:
if isinstance(val, set):
val = val
elif isinstance(val, (list, tuple)):
val = set(val)
else:
val = set([val])
val = manager.SetVertexPropertyManager(
data_type,
vertex_prop,
card,
{vertex_prop(data_type, val=self.validate(v), card=card)
for v in val})
else:
val = vertex_prop(data_type, val=self.validate(val))
return val
......
# Copyright 2016 ZEROFAIL
#
# This file is part of Goblin.
#
# Goblin is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# Goblin is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with Goblin. If not, see <http://www.gnu.org/licenses/>.
"""Enum to handle property cardinality"""
import enum
class Cardinality(enum.Enum):
"""Property cardinality"""
single = 1
list = 2
set = 3
......@@ -19,9 +19,7 @@ import logging
import inflection
from goblin import abc
from goblin import mapper
from goblin import properties
from goblin import abc, cardinality, exception, mapper, properties
logger = logging.getLogger(__name__)
......@@ -34,14 +32,22 @@ class ElementMeta(type):
:py:class:`property.Property` with :py:class:`property.PropertyDescriptor`.
"""
def __new__(cls, name, bases, namespace, **kwds):
if bases:
namespace['__type__'] = bases[0].__name__.lower()
if name == 'VertexProperty':
element_type = name.lower()
elif bases:
element_type = bases[0].__name__.lower()
else:
element_type = name.lower()
namespace['__type__'] = element_type
if not namespace.get('__label__', None):
namespace['__label__'] = inflection.underscore(name)
props = {}
new_namespace = {}
for k, v in namespace.items():
if isinstance(v, abc.BaseProperty):
if element_type == 'edge' and hasattr(v, 'cardinality'):
raise exception.MappingError(
'Edge property cannot have set/list cardinality')
props[k] = v
v = v.__descriptor__(k, v)
new_namespace[k] = v
......@@ -110,7 +116,7 @@ class GenericEdge(Edge):
Class used to build edges when user defined edges class is not available.
Generally not instantiated by end user.
"""
pass
class VertexPropertyDescriptor:
"""
......@@ -119,45 +125,46 @@ class VertexPropertyDescriptor:
"""
def __init__(self, name, vertex_property):
self._prop_name = name
self._name = '_' + name
self._vertex_property = vertex_property.__class__
self._data_type = vertex_property.data_type
self._default = vertex_property.default
self._cardinality = vertex_property._cardinality
def __get__(self, obj, objtype):
if obj is None:
return self._vertex_property
return getattr(objtype.__mapping__, self._prop_name)
default = self._default
if default:
default = self._data_type.validate(default)
default = self._vertex_property(self._default)
default = self._data_type.validate_vertex_prop(
default, self._cardinality, self._vertex_property,
self._data_type)
return getattr(obj, self._name, default)
def __set__(self, obj, val):
if isinstance(val, (list, tuple , set)):
vertex_property = []
for v in val:
v = self._data_type.validate(v)
vertex_property.append(
self._vertex_property(self._data_type, value=v))
else:
val = self._data_type.validate(val)
vertex_property = self._vertex_property(self._data_type, value=val)
setattr(obj, self._name, vertex_property)
if val:
val = self._data_type.validate_vertex_prop(
val, self._cardinality, self._vertex_property, self._data_type)
setattr(obj, self._name, val)
class VertexProperty(Element, abc.BaseProperty):
"""Base class for user defined vertex properties. Not yet supported."""
class VertexProperty(Vertex, abc.BaseProperty):
"""Base class for user defined vertex properties."""
__descriptor__ = VertexPropertyDescriptor
def __init__(self, data_type, *, value=None, default=None):
def __init__(self, data_type, *, val=None, default=None, db_name=None,
card=None):
if isinstance(data_type, type):
data_type = data_type()
self._data_type = data_type
self._value = value
self._val = val
self._default = default
self._db_name = db_name
if card is None:
card = cardinality.Cardinality.single
self._cardinality = card
@property
def default(self):
......@@ -169,7 +176,15 @@ class VertexProperty(Element, abc.BaseProperty):
@property
def value(self):
return self._value
return self._val
@property
def db_name(self):
return self._db_name
@property
def cardinality(self):
return self._cardinality
def __repr__(self):
return '<{}(type={}, value={})'.format(self.__class__.__name__,
......
"""Managers for multi cardinality vertex properties"""
class VertexPropertyManager:
def __init__(self, data_type, vertex_prop, card):
self._data_type = data_type
self._vertex_prop = vertex_prop
self._card = card
self._mapper_func = vertex_prop.__mapping__.mapper_func
@property
def mapper_func(self):
return self._mapper_func
def __call__(self, val):
results = []
for v in self:
if v.value == val:
results.append(v)
if len(results) == 1:
results = results[0]
elif not results:
results = None
return results
class ListVertexPropertyManager(list, VertexPropertyManager):
def __init__(self, data_type, vertex_prop, card, obj):
VertexPropertyManager.__init__(self, data_type, vertex_prop, card)
list.__init__(self, obj)
def append(self, val):
val = self._data_type.validate(val)
val = self._vertex_prop(self._data_type, val=val, card=self._card)
super().append(val)
class SetVertexPropertyManager(set, VertexPropertyManager):
def __init__(self, data_type, vertex_prop, card, obj):
VertexPropertyManager.__init__(self, data_type, vertex_prop, card)
set.__init__(self, obj)
def add(self, val):
val = self._data_type.validate(val)
val = self._vertex_prop(self._data_type, val=val, card=self._card)
super().add(val)
......@@ -28,31 +28,84 @@ logger = logging.getLogger(__name__)
def map_props_to_db(element, mapping):
"""Convert OGM property names/values to DB property names/values"""
property_tuples = []
props = mapping.properties
props = mapping.ogm_properties
for ogm_name, (db_name, data_type) in props.items():
val = getattr(element, ogm_name, None)
property_tuples.append((db_name, data_type.to_db(val)))
if val and isinstance(val, (list, set)):
card = None
for v in val:
metaprops = get_metaprops(v, v.__mapping__)
property_tuples.append(
(card, db_name, data_type.to_db(v.value), metaprops))
card = v.cardinality
else:
if hasattr(val, '__mapping__'):
metaprops = get_metaprops(val, val.__mapping__)
val = val.value
else:
metaprops = None
property_tuples.append(
(None, db_name, data_type.to_db(val), metaprops))
return property_tuples
def get_metaprops(vertex_property, mapping):
props = mapping.ogm_properties
metaprops = {}
for ogm_name, (db_name, data_type) in props.items():
val = getattr(vertex_property, ogm_name, None)
metaprops[db_name] = data_type.to_db(val)
return metaprops
def map_vertex_to_ogm(result, element, *, mapping=None):
"""Map a vertex returned by DB to OGM vertex"""
for db_name, value in result['properties'].items():
# This will be more complex for vertex properties...
value = value[0]['value']
name, data_type = mapping.properties.get(db_name, (db_name, None))
metaprop_dict = {}
if len(value) > 1:
values = []
for v in value:
values.append(v['value'])
metaprops = v.get('properties', None)
if metaprops:
metaprop_dict[v['value']] = metaprops
value = values
else:
metaprops = value[0].get('properties', None)
value = value[0]['value']
if metaprops:
metaprop_dict[value] = metaprops
name, data_type = mapping.db_properties.get(db_name, (db_name, None))
if data_type:
value = data_type.to_ogm(value)
setattr(element, name, value)
if metaprop_dict:
vert_prop = getattr(element, name)
vert_prop.mapper_func(metaprop_dict, vert_prop)
setattr(element, '__label__', result['label'])
setattr(element, 'id', result['id'])
return element
def map_vertex_property_to_ogm(result, element, *, mapping=None):
"""Map a vertex returned by DB to OGM vertex"""
for val, metaprops in result.items():
if isinstance(element, (list, set)):
current = element(val)
else:
current = element
for db_name, value in metaprops.items():
name, data_type = mapping.db_properties.get(
db_name, (db_name, None))
if data_type:
value = data_type.to_ogm(value)
setattr(current, name, value)
def map_edge_to_ogm(result, element, *, mapping=None):
"""Map an edge returned by DB to OGM edge"""
for db_name, value in result.get('properties', {}).items():
name, data_type = mapping.properties.get(db_name, (db_name, None))
name, data_type = mapping.db_properties.get(db_name, (db_name, None))
if data_type:
value = data_type.to_ogm(value)
setattr(element, name, value)
......@@ -85,14 +138,22 @@ def _check_id(rid, eid):
# DB <-> OGM Mapping
def create_mapping(namespace, properties):
"""Constructor for :py:class:`Mapping`"""
element_type = namespace.get('__type__', None)
if element_type:
if element_type == 'vertex':
mapping_func = map_vertex_to_ogm
return Mapping(namespace, element_type, mapping_func, properties)
elif element_type == 'edge':
mapping_func = map_edge_to_ogm
return Mapping(namespace, element_type, mapping_func, properties)
element_type = namespace['__type__']
if element_type == 'vertex':
mapping_func = map_vertex_to_ogm
mapping = Mapping(
namespace, element_type, mapping_func, properties)
elif element_type == 'edge':
mapping_func = map_edge_to_ogm
mapping = Mapping(
namespace, element_type, mapping_func, properties)
elif element_type == 'vertexproperty':
mapping_func = map_vertex_property_to_ogm
mapping = Mapping(
namespace, element_type, mapping_func, properties)
else:
mapping = None
return mapping
class Mapping:
......@@ -104,7 +165,8 @@ class Mapping:
self._label = namespace['__label__']
self._element_type = element_type
self._mapper_func = functools.partial(mapper_func, mapping=self)
self._properties = {}
self._db_properties = {}
self._ogm_properties = {}
self._map_properties(properties)
@property
......@@ -118,13 +180,18 @@ class Mapping:
return self._mapper_func
@property
def properties(self):
def db_properties(self):
"""A dictionary of property mappings"""
return self._db_properties
@property
def ogm_properties(self):
"""A dictionary of property mappings"""
return self._properties
return self._ogm_properties
def __getattr__(self, value):
try:
mapping, _ = self._properties[value]
mapping, _ = self._ogm_properties[value]
return mapping
except:
raise exception.MappingError(
......@@ -138,10 +205,14 @@ class Mapping:
db_name = prop.db_name
else:
db_name = '{}__{}'.format(self._label, name)
self._properties[db_name] = (name, data_type)
self._properties[name] = (db_name, data_type)
if hasattr(prop, '__mapping__'):
if not self._element_type == 'vertex':
raise exception.MappingError(
'Only vertices can have vertex properties')
self._db_properties[db_name] = (name, data_type)
self._ogm_properties[name] = (db_name, data_type)
def __repr__(self):
return '<{}(type={}, label={}, properties={})>'.format(
self.__class__.__name__, self._element_type, self._label,
self._properties)
self._ogm_properties)
......@@ -15,11 +15,11 @@
# You should have received a copy of the GNU Affero General Public License
# along with Goblin. If not, see <http://www.gnu.org/licenses/>.
"""Classes to handle proerties and data type definitions"""
"""Classes to handle properties and data type definitions"""
import logging
from goblin import abc, exception
from goblin import abc, cardinality, exception
logger = logging.getLogger(__name__)
......@@ -43,7 +43,8 @@ class PropertyDescriptor:
return getattr(obj, self._name, self._default)
def __set__(self, obj, val):
setattr(obj, self._name, self._data_type.validate(val))
val = self._data_type.validate(val)
setattr(obj, self._name, val)
def __delete__(self, obj):
# hmmm what is the best approach here
......@@ -110,7 +111,7 @@ class Integer(abc.DataType):
if val is not None:
try:
return int(val)
except ValueError as e:
except (ValueError, TypeError) as e:
raise exception.ValidationError(
'Not a valid integer: {}'.format(val)) from e
......
......@@ -195,7 +195,7 @@ class Session(connection.AbstractConnection):
del edge
return result
async def save(self, element):
async def save(self, elem):
"""
Save an element to the db.
......@@ -203,13 +203,13 @@ class Session(connection.AbstractConnection):
:returns: :py:class:`Element<goblin.element.Element>` object
"""
if element.__type__ == 'vertex':
result = await self.save_vertex(element)
elif element.__type__ == 'edge':
result = await self.save_edge(element)
if elem.__type__ == 'vertex':
result = await self.save_vertex(elem)
elif elem.__type__ == 'edge':
result = await self.save_edge(elem)
else:
raise exception.ElementError(
"Unknown element type: {}".format(element.__type__))
"Unknown element type: {}".format(elem.__type__))
return result
async def save_vertex(self, vertex):
......@@ -222,7 +222,7 @@ class Session(connection.AbstractConnection):
"""
result = await self._save_element(
vertex, self._check_vertex,
self.traversal_factory.add_vertex,
self._add_vertex,
self.update_vertex)
self.current[result.id] = result
return result
......@@ -240,7 +240,7 @@ class Session(connection.AbstractConnection):
"Edges require both source/target vertices")
result = await self._save_element(
edge, self._check_edge,
self.traversal_factory.add_edge,
self._add_edge,