Are people allowed to take visitation stones?

I’m not Jewish but many chinese families visit the cemetery this time of year to see their late relatives “QingMing”; give them “gifts” (we burn paper replicas of clothes; phones, cars, to send up to heaven). We also eat a picnic with every grave we visit. Growing up; we would take stones we find on nearby graves to hold down the picnic blanket. I just recently found out what visitation stones were for – none of us realized they had any religious or cultural meaning. Are taking visitation stones taboo? Can we borrow them and put them back on the grave we took them for? Or should we avoid touching them in general?