Apply Scene Detection with Graphs

Application of knowledge graph built from Wiki data dump

Continuing the last article, you might ask, how do we utilize the Wiki data dump into Dgraph? I’d like to walk through how to use graphs we built last time to do scene detection. (What is scene detection?check out this paper here https://arxiv.org/pdf/1812.05736.pdf)

Before that, you have to first take the videos, get every frame of the video, and run them through object detection to get out a list of objects. Scene detection is to find out only the objects appearing together that makes sense, and use those as our context of the video for classification.

Building the graph

As mentioned, we will insert wiki entities based on their categories because that is the most structure thing you can find on any wiki page. The way I construct the hierarchy is by going up two levels up their parent categories so the hierarchy is clean and can link to more general concepts.

To build the graph, I start with Open image relationship csv file to insert (Subject, relationship, Object) into the graph (e.g. Woman holds handbag) and expand out the entities with wiki data parent categories (e.g. handbag -> fashion -> fashion industry). complete code here

Another part is to expand entities from wiki vital articles (code here) to expand the graph with everyday objects and their parent categories. Hopefully this gives us a good base in the dgraph to query common objects and get out meaningful relationships.

In conclusion, To build the graph, run:

python3 pipeline.py

Querying the graph

To query the graph, run:

python3 query_graph.py

What it is doing behind the scenes, is querying the relationship between two detected objects. For example, given “Woman” and “Handbag”, the graph will return “holds, wears” and together, “Woman holds Handbag” will create a scene such as shopping or vacation. If two objects detected is not connected with such relationships, the graph will query the lowest common ancestor of the two objects with code here and return whether the two objects has common ancestor within 3 levels traversed upward.

Below is a sample query that you can try out in your dgraph:

{
  everyone(func: allofterms(name, “Dessert”))
  @recurse (depth:4, loop:true)
  {
    name
    is_page_in (first: 5)
    ~is_page_in (first: 5)
    has_subcategories (first: 5)
    ~has_subcategories (first:5)
  }
}

To Be Continued:
- Link prediction on the node’s relationship/train GNN,
- Expand wiki data relationships with ConceptNet,
- Enhance knowledge graph as prediction engine like Google search engine (upgrade/ downgrade certain model predictions and stitch results into one final output)