Skip to content

Commit 934feb2

Browse files
committed
yugabyte#2101: Fixed race between Tablet::Flush and Tablet::Shutdown
Summary: ``` WARNING: ThreadSanitizer: data race (pid=11311) 1762 Read of size 8 at 0x7b74000cfb58 by thread T155: 1763 #0 std::__1::unique_ptr<rocksdb::DB, std::__1::default_delete<rocksdb::DB> >::operator bool() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20200829090443-f431681041-centos/installed/tsan/libcxx/include/c++/v1/memory:2619:19 (libtablet.so+0x211067) 1764 #1 yb::tablet::Tablet::Flush(yb::tablet::FlushMode, yb::tablet::FlushFlags, long) src/yb/tablet/tablet.cc:1847 (libtablet.so+0x211067) 1765 #2 yb::tserver::MiniTabletServer::FlushTablets(yb::tablet::FlushMode, yb::tablet::FlushFlags)::$_2::operator()(yb::tablet::TabletPeer*) const src/yb/tserver/mini_tablet_server.cc:200:35 (libtserver.so+0x1a0997) ... 1771 #8 yb::tserver::MiniTabletServer::FlushTablets(yb::tablet::FlushMode, yb::tablet::FlushFlags) src/yb/tserver/mini_tablet_server.cc:196:10 (libtserver.so+0x19f4e8) 1772 #9 yb::MiniCluster::FlushTablets(yb::tablet::FlushMode, yb::tablet::FlushFlags) src/yb/integration-tests/mini_cluster.cc:369:5 (libintegration-tests.so+0x10cf88) 1773 #10 yb::client::QLStressTest_LongRemoteBootstrap_Test::TestBody()::$_8::operator()() const src/yb/client/ql-stress-test.cc:972:7 (ql-stress-test+0x4f2f1d) Previous write of size 8 at 0x7b74000cfb58 by thread T49 (mutexes: write M263877791325157728): 1780 #0 std::__1::unique_ptr<rocksdb::DB, std::__1::default_delete<rocksdb::DB> >::reset(rocksdb::DB*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20200829090443-f431681041-centos/installed/tsan/libcxx/include/c++/v1/memory:2632:20 (libtablet.so+0x209c9c) 1781 #1 yb::tablet::ResetRocksDB(bool, rocksdb::Options const&, std::__1::unique_ptr<rocksdb::DB, std::__1::default_delete<rocksdb::DB> >*) src/yb/tablet/tablet.cc:950 (libtablet.so+0x209c9c) 1782 #2 yb::tablet::Tablet::ResetRocksDBs(yb::StronglyTypedBool<yb::tablet::Destroy_Tag>, yb::StronglyTypedBool<yb::tablet::DisableFlushOnShutdown_Tag>) src/yb/tablet/tablet.cc:967:27 (libtablet.so+0x209b3b) 1783 #3 yb::tablet::Tablet::CompleteShutdown(yb::StronglyTypedBool<yb::tablet::IsDropTable_Tag>) src/yb/tablet/tablet.cc:931:3 (libtablet.so+0x205bac) 1784 #4 yb::tablet::TabletPeer::CompleteShutdown(yb::StronglyTypedBool<yb::tablet::IsDropTable_Tag>) src/yb/tablet/tablet_peer.cc:475:14 (libtablet.so+0x287f71) 1785 #5 yb::tablet::TabletPeer::Shutdown(yb::StronglyTypedBool<yb::tablet::IsDropTable_Tag>) src/yb/tablet/tablet_peer.cc:529:5 (libtablet.so+0x28896d) 1786 #6 yb::tserver::TSTabletManager::DeleteTablet(string const&, yb::tablet::TabletDataState, boost::optional<long> const&, boost::optional<yb::tserver::TabletServerErrorPB_Code>*) src/yb/tserver/ts_tablet_manager.cc:1297:16 (libtserver.so+0x22d59e) 1787 #7 yb::tserver::TabletServiceAdminImpl::DeleteTablet(yb::tserver::DeleteTabletRequestPB const*, yb::tserver::DeleteTabletResponsePB*, yb::rpc::RpcContext) src/yb/tserver/tablet_service.cc:1158:41 (libtserver.so+0x1e1de7) 1788 #8 yb::tserver::TabletServerAdminServiceIf::Handle(std::__1::shared_ptr<yb::rpc::InboundCall>) src/yb/tserver/tserver_admin.service.cc:130:7 (libtserver_admin_proto.so+0x73cb0) 1789 #9 yb::rpc::ServicePoolImpl::Handle(std::__1::shared_ptr<yb::rpc::InboundCall>) src/yb/rpc/service_pool.cc:262:19 (libyrpc.so+0x20aa57) 1790 #10 yb::rpc::InboundCall::InboundCallTask::Run() src/yb/rpc/inbound_call.cc:212:13 (libyrpc.so+0x1745ee) ``` Added missing `ScopedRWOperation` into `Tablet::Flush` to avoid destroying tablets during flush. Test Plan: `ybd --remote --dltp tsan --cxx-test client_ql-stress-test --gtest_filter QLStressTest.LongRemoteBootstrap -n 500 -- -p 1` Reviewers: bogdan, mikhail Reviewed By: mikhail Subscribers: zyu, ybase Differential Revision: https://phabricator.dev.yugabyte.com/D9541
1 parent a9d7b5b commit 934feb2

File tree

1 file changed

+2
-0
lines changed

1 file changed

+2
-0
lines changed

src/yb/tablet/tablet.cc

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1836,6 +1836,8 @@ void Tablet::AcquireLocksAndPerformDocOperations(std::unique_ptr<WriteOperation>
18361836
Status Tablet::Flush(FlushMode mode, FlushFlags flags, int64_t ignore_if_flushed_after_tick) {
18371837
TRACE_EVENT0("tablet", "Tablet::Flush");
18381838

1839+
ScopedRWOperation pending_op(&pending_op_counter_);
1840+
18391841
rocksdb::FlushOptions options;
18401842
options.ignore_if_flushed_after_tick = ignore_if_flushed_after_tick;
18411843
bool flush_intents = intents_db_ && HasFlags(flags, FlushFlags::kIntents);

0 commit comments

Comments
 (0)