What’s the best way to keep track (model/db architeture-wise) of unprocessed model combinations in Ruby on Rails?

Within Site1:

User has_and_belongs_to_many :rollers

Roller has_and_belongs_to_many :users

Roller has_many :tweets

There is a site “over there” (Site2) that has an API that I need to send information to. I don’t want to send the same information over and over to that API. After Site1 grows, there could be many tens of thousands of objects that have to get sent to the foreign API, and that would start to take a long time, and cause undue processing on the foreign API engine.

For each User‘s Rollers, I want to send all of their Tweets over to that API. But what is the best way to keep track of which User/Roller/Tweet combo I’ve already sent, so that I can easily scope something to, effectively, unsent_tweets?

  1. At this moment there is only one API we interact with and the User (of Site1) is actually not important to know for sending the data. In other words, it’s really the roller/tweet combos that need to be tracked. But maybe someday in the future, we might need to call different endpoints for different users. So it probably doesn’t hurt to store the combination of User, Roller, and Tweet, and even the endpoint called. But let’s stick to User/Roller/Tweet for now.

  2. There could be a large number of users with the same exact rollers and tweets. For example, lots of people might be following a celebrity. Storing some kind of JSON/structured data in a field on User or Roller to keep track of what’s been done could result in some really tremendous and nasty stuff.

  3. A dedicated model/object to store these is probably the right way? If Tweet has_many :api_tracking_combinations then I could have the APITrackingCombination model store a user_id and a roller_id. This could still result in a pretty massive table in the long-run.

    It’s possible to scope with variables, but I’m trying to find tweets where the user/roller combo has not yet been processed. In other words, I’d want to find tweets where there is NOT an instance of the combination object.

If I went down the path of #3 I am not really sure how to implement the query-ing. This problem seems to be either a ton of looping over stuff (which I can probably figure out how to code but seems wildly expensive/inefficient) or some kind of crazy database query magic that I’m incapable of understanding (which might be expensive on the database).

What do you think is the best solution here?


A cli generate component For React.js

Samsung Galaxy S22+ Renders Surface Online, Provides Key Specifications