User has_and_belongs_to_many :rollers
Roller has_and_belongs_to_many :users
Roller has_many :tweets
There is a site “over there” (Site2) that has an API that I need to send information to. I don’t want to send the same information over and over to that API. After Site1 grows, there could be many tens of thousands of objects that have to get sent to the foreign API, and that would start to take a long time, and cause undue processing on the foreign API engine.
Rollers, I want to send all of their
Tweets over to that API. But what is the best way to keep track of which User/Roller/Tweet combo I’ve already sent, so that I can easily scope something to, effectively,
At this moment there is only one API we interact with and the User (of Site1) is actually not important to know for sending the data. In other words, it’s really the roller/tweet combos that need to be tracked. But maybe someday in the future, we might need to call different endpoints for different users. So it probably doesn’t hurt to store the combination of User, Roller, and Tweet, and even the endpoint called. But let’s stick to User/Roller/Tweet for now.
There could be a large number of users with the same exact rollers and tweets. For example, lots of people might be following a celebrity. Storing some kind of JSON/structured data in a field on
Rollerto keep track of what’s been done could result in some really tremendous and nasty stuff.
A dedicated model/object to store these is probably the right way? If
Tweet has_many :api_tracking_combinationsthen I could have the
APITrackingCombinationmodel store a
roller_id. This could still result in a pretty massive table in the long-run.
It’s possible to scope with variables, but I’m trying to find tweets where the user/roller combo has not yet been processed. In other words, I’d want to find tweets where there is NOT an instance of the combination object.
If I went down the path of #3 I am not really sure how to implement the query-ing. This problem seems to be either a ton of looping over stuff (which I can probably figure out how to code but seems wildly expensive/inefficient) or some kind of crazy database query magic that I’m incapable of understanding (which might be expensive on the database).
What do you think is the best solution here?