Quantcast
Channel: Apache Timeline
Viewing all articles
Browse latest Browse all 5648

avdl schema compatibility

$
0
0
Hi All, I am trying to use avro's protocol definition to provide the basis for an RPC system. In this case, I'll be handling the handshaking and transport layer rather the avro's IPC package. Are the schemas generated from the protocol files compatible with the standard DataFileWriter/Reader schemas (I am using python3)? If I try to load a file generated from an avdl message (using DataFileWriter(bytes, DatumWriter(), protocol.message_map['hash'].request)) then when I try to load it, I receive error messages stating that the writer_schema is essentially invalid (see stacktrace below).

Here is a code snippet that seems to show an invalid schema being generated. I think its because the value of the 'type' field (for objfsptr) is a python/json dictionary rather that the 'record' type i think its supposed to be.

import avro.protocol
p = avro.protocol.Parse(open("service.avpr").read())
print(p.message_map['hash'].request.to_json())

[{'name': 'objfsptr',
'type': {'fields': [{'name': 'uri', 'type': 'string'}],
'name': 'ObjFSPtr',
'namespace': 'FooServiceType',
'type': 'record'}},
{'default': 10, 'name': 'x', 'type': 'int'}]

Stacktrace from a deserialization attempt:

Traceback (most recent call last):
File "./schema_test.py", line 74, in <module>
requests = deserializer.deserialize('hash', srequest)
File "/data/bluecoat/services.py", line 123, in deserialize
reader = DataFileReader(io.BytesIO(raw_bytes), DatumReader())
File "/data/avro/datafile.py", line 363, in __init__
schema.Parse(self.GetMeta(SCHEMA_KEY).decode('utf-8')))
File "/data/avro/schema.py", line 1283, in Parse
return SchemaFromJSONData(json_data, names)
File "/data/avro/schema.py", line 1254, in SchemaFromJSONData
return parser(json_data, names=names)
File "/data/avro/schema.py", line 1142, in _SchemaFromJSONArray
return UnionSchema(map(MakeSchema, json_array))
File "/data/avro/schema.py", line 866, in __init__
self._schemas = tuple(schemas)
File "/data/avro/schema.py", line 1141, in MakeSchema
return SchemaFromJSONData(json_data=desc, names=names)
File "/data/avro/schema.py", line 1254, in SchemaFromJSONData
return parser(json_data, names=names)
File "/data/avro/schema.py", line 1154, in _SchemaFromJSONObject
if type in PRIMITIVE_TYPES:
TypeError: unhashable type: 'dict'

service.avdl

@namespace("FooService")
protocol MyProtocol {

@namespace("FooServiceType")
record ObjFSPtr {
string uri;

string hash(FooServiceType.ObjFSPtr objfsptr, int x=10);

generated service.avpr:

"protocol" : "MyProtocol",
"namespace" : "FooService",
"types" : [ {
"type" : "record",
"name" : "ObjFSPtr",
"namespace" : "FooServiceType",
"fields" : [ {
"name" : "uri",
"type" : "string"
} ]
} ],
"messages" : {
"hash" : {
"request" : [ {
"name" : "objfsptr",
"type" : "FooServiceType.ObjFSPtr"
}, {
"name" : "x",
"type" : "int",
"default" : 10
} ],
"response" : "string"

Thanks!

Jonathan Camp

Viewing all articles
Browse latest Browse all 5648

Trending Articles