Around 13,000 British nationals visit Brunei every year (source: Brunei Immigration). Most visits are trouble free.
There is a low threat from terrorism. See Terrorism.
Crime levels are low, but there are occasional incidents of petty crime against tourists. See Crime.
Take out comprehensive travel and medical insurance before you travel.