Unless you can properly keep the Sabbath while visiting them--engaging in the same spiritual conversation and fellowship that you would with others--I would not do this. If you really want to catch up, you can meet on another day of the week.
One technique that can help even non-Sabbatarians think about holy things is to sit down and have a sermon review. Have everyone sit down and ask each person to remember a point or two from the worship service (and especially the sermon). It can help stir spiritual conversation and be a starting point for it.
It is likely that the non-Sabbatarian is not used to having a spiritual frame of mind, thinking about heavenly things, or speaking about them. In my experience, it is only the training that Sabbath-keeping provides that tends to cause people to be that way. It isn't always the case, but it often is. If this is the case, you will have to take more leadership and initiative to try to keep things focused on heavenly things--it will be a ministry opportunity more so than a visit of equals. If these things are the case, and you aren't up for it, I wouldn't go.
It would be important to communicate to the non-Sabbatarian friend that you keep the Sabbath because it is a delight and commanded by God and that is how you will keep it if you visit with them. You can spin it in a positive way if you do go: "Would you like to give Sabbath keeping a try while I'm there? We try to keep conversation heavenly minded? Maybe we'll sing some psalms or pray together?"
