问题:
我在ES里面建了两个一样结构的订单索引,一个放今日的实时数据,一个放昨日之前的日结数据,我要怎么才能获得两个索引相同订单记录的不同之处呢?
例子:
今天实时我存的一个订单本来应该配达了,但是因为某些异常原因没有更新。而日结任务从订单源表拉取的订单数据是正确的。我应该怎样快速比对两个索引,并知道这些数据不一致的文档呢?
思路:
我写一个一个dsl,方法比较笨,就是用terms聚合同时查询两个索引,过滤掉文档数大于1的,就是数据不一致的订单,但是感觉太笨重了,效率也不高,不知道各位大佬有没有更好的方法
以下是DSL语句:
GET perform_tao_order_realtime,perform_tao_order_gather/_search?size=0
{"query":{"bool":{"must":[{"term":{"deliveryStatus":{"value":"0"}}},{"range":{"waveOrderCreatetime":{"gte":"2019-11-19 00:00:00","lte":"2019-11-19 23:59:59"}}}]}},"aggs":{"order_terms":{"terms":{"script":{"source":"params['_source']['storeNo'] +'_' + params['_source']['waveOrderNo'] +'_' + params['_source']['orderNo'] +'_' + doc['productPcs'].value +'_' + doc['insideConsumingTime'].value +'_' + doc['takeConsumingTime'].value","lang":"painless"},"order":{"_count":"desc"},"size":1000000},"aggs":{"distinct_id":{"value_count":{"field":"orderNo"}},"count_bucket_selector":{"bucket_selector":{"buckets_path":{"orderTerms":"distinct_id"},"script":"params.orderTerms < 2"}}}},"totalNum":{"sum_bucket":{"buckets_path":"order_terms>distinct_id"}}}}
我在ES里面建了两个一样结构的订单索引,一个放今日的实时数据,一个放昨日之前的日结数据,我要怎么才能获得两个索引相同订单记录的不同之处呢?
例子:
今天实时我存的一个订单本来应该配达了,但是因为某些异常原因没有更新。而日结任务从订单源表拉取的订单数据是正确的。我应该怎样快速比对两个索引,并知道这些数据不一致的文档呢?
思路:
我写一个一个dsl,方法比较笨,就是用terms聚合同时查询两个索引,过滤掉文档数大于1的,就是数据不一致的订单,但是感觉太笨重了,效率也不高,不知道各位大佬有没有更好的方法
以下是DSL语句:
GET perform_tao_order_realtime,perform_tao_order_gather/_search?size=0
{"query":{"bool":{"must":[{"term":{"deliveryStatus":{"value":"0"}}},{"range":{"waveOrderCreatetime":{"gte":"2019-11-19 00:00:00","lte":"2019-11-19 23:59:59"}}}]}},"aggs":{"order_terms":{"terms":{"script":{"source":"params['_source']['storeNo'] +'_' + params['_source']['waveOrderNo'] +'_' + params['_source']['orderNo'] +'_' + doc['productPcs'].value +'_' + doc['insideConsumingTime'].value +'_' + doc['takeConsumingTime'].value","lang":"painless"},"order":{"_count":"desc"},"size":1000000},"aggs":{"distinct_id":{"value_count":{"field":"orderNo"}},"count_bucket_selector":{"bucket_selector":{"buckets_path":{"orderTerms":"distinct_id"},"script":"params.orderTerms < 2"}}}},"totalNum":{"sum_bucket":{"buckets_path":"order_terms>distinct_id"}}}}
0 个回复