Data Ingest Tool
Empower Self-Service: from days to minutes
Last updated
Empower Self-Service: from days to minutes
Last updated
Data ingest is a tool to empower the internal field team and marketing analyst ingest ad-hoc data table without writing scripts.
Being able to quickly ingest new or update data sources to a centralized platform is key for marketers to make informed marketing decisions in a timely manner. As the data hub for our clients, ActionIQ ingested stream and batch data frequently to ensure everything is up to date. The majority of the ingest work was setup during the on-boarding period (3–6 months).
As our customers getting more used to the platform, new use cases emerge which usually involve more ad-hoc data tables.
Previously, the entire ingest workflow was done outside the core platform, it requires strong technical skills and usually takes several days of work our field & client’ marketing teams since there’s a lot of back and forth.
This is problematic when a marketer wants to quickly ingest or test out a new data source, for example, the output of a LTV or churn prediction model.
I own the design of this project, from initial research, prototyping, visual design to usability testing. I collaborated with 1 product manager and 2 engineers to scope and ship and MVP version.
Before this project, I have little idea what data ingest process looks like. New terms were all over the place. Therefore, I conducted a series of internal interviews to better understand the problem space:
Typical user: field engineers —> what is the normal workflow to ingest a data table?
Performance expert: backend engineers —> how smart could we be, what's the limitations?
Roadmap master: product managers —> how to scope out the MVP with a bigger roadmap in mind?
Those interviews were extremely helpful for me to find a focus, understand the pain points and be aware of the constraints.
The deliverables out of this initial stage were:
A chart to visualize the complete workflow (jobs) and user story (feelings) so that I could keep the big picture in mind
A low-fi prototype to visualize the MVP scope to make sure the team are on the same page
After the conversations, I synthesized the information and came up with 2 major goals
Design-wise
How might we provide just-enough in-platform guidance so that both internal and external users could feel well-prepared as they ingest data tables?
Product-wise
How might we scope out MVP to balance the customer use case needs, engineering resource and performance limitation without loosing the long-term roadmap vision?
Quick prototyping is especially useful when it comes to an area you are not super familiar with —when some important details/concerns would not emerge until you show it to a wider audience.
I built out a prototype quickly right after the kick-off meeting and reviewed it with the product manager to make sure it covers the PRD requirements. I kept updating the prototype and discovered something new after each interview.
Built multiple versions before finalizing the flow
V1: Requirement validation: “Does it cover everything we need”
→ Deliverable: visualize the requirements in the “ideal” world
V2: Feedback: “It’ll take a long time to calculate, better review the type beforehand”
→ Iteration: separate the "review" and "define ingest rule" into 2 steps
V3: Feedback: “Am I actually deleting the rows right now? That would be scary.”
→ Iteration: define rules, not action
V4: Feedback: “What would you recommend?”
→ Iteration: make highest quality dataset by default
V5: Feedback “Should I wait on the page?”
→ Iteration: Come back later
After finalizing the flow, I designed the hi-fi prototype based on the existing patterns we have as well as envisioning improvements we could introduce to the rest of the platform in the future.
This is the strategy I tried out after the lessons learned from previous projects — — and it worked out perfectly!
Even though the best scenario is to have a clear requirement list before diving into design, it never happens. Requirements are changing all the time, especially for start-ups. Even worse, people sometimes make random decisions at a random meeting without good reasoning and this is likely to bring some old discussions back at a very late stage.
It can be very frustrating for designers — — especially you assume everyone’s on the same page and the hi-fi prototype is so ready to go.
For this project, I kept a doc to document the requirement changes, reasoning behind the changes and corresponding prototype updates. It gave me more confidence when making design decisions. It also made me more conscious and selective about all the feedback I received.
Even though data ingest dashboard is a stand-alone feature, it is still important to make the hi-fi style consistent with the rest of the product. I utilized the style-guide as much as possible and tried to build new widgets that could align with the general style.
Quickly adopted by most of the clients with 30+ new data tables ingested within the first 3 months
Leadership team started considering make ingesting tool from a “internal beta”to a separate product line
